It is known that conventional computer storage Backup apparatus deduplicate files by recognizing hashes of shards of binary objects. It is known that backup services operating off a wide area network use pattern recognition to de-dup data transfer over the Internet. Cloud-based storage services grow at increasingly rapid rates. Improved optimization is needed to scale with demand. Within this patent application we define a shard to be a portion of a binary object. An example of a shard is a one megabyte or less segment of a data file.
Log—Structured file systems have been proposed, evaluated, and deprecated as proposed architectures for file systems. Rosenblum & Ousterhout (1992), ACM Transactions on Computer Systems, vol. 10, no. 1, pp. 26-52, Association for Computing Machinery, February 1992 and Seltzer, et al. (1995). These techniques are insufficient to optimize backup performance.
What is needed is a way to increase the performance of a backup system with very large workload submitted by many remote clients. Backup systems must themselves have effective backup in order to ensure that they are quickly available when the call for client restoration occurs. What is needed is reduced latency for restoring a backup store per se so that it is highly available for its clients.